Implement epoll APIs in the JS filesystem#27207
Conversation
aacb3d7 to
57e98fb
Compare
sbc100
left a comment
There was a problem hiding this comment.
I think this like this direction.
I've not had time to look at all the details yet, but it seems like a great idea to unify the node events like this.
0e64f2b to
9b46f69
Compare
97a4580 to
d33958b
Compare
|
I'm liking the direction of this commit but still a little overwhelmed by the size of it. Can you think of any more ways to split it up? I'm not sure myself... for example, could we land the internal refactoring of the poll/select to use the new notification system before we land the rest of epoll? Maybe not practically separable? |
I've refactored out the JS notification changes into #27226 if that helps? |
…27226) This refactors out a partial base from #27207. Move the readiness wait-queue onto FSNode (addListener/notifyListeners) so dup'd fds share one queue, and rewrite the suspending poll() path (readPollfds/writePollfds/pollWait) on top of it. Producers (SOCKFS, PIPEFS) now notify the node on readiness transitions, and close() wakes waiters with POLLNVAL. - FSNode gains `addListener(cb, exclusive)` / `notifyListeners(flags)` — a per-inode Set of listener entries. It lives on the node (not the fd) so dup'd fds share one queue. Only nodes with real readiness (sockets, pipes) ever populate it; always-ready types (regular files, ttys) never touch it. - The suspending `poll()` path is rewritten on top of it (`readPollfds`/`writePollfds`/`pollWait`/`pollOne`), replacing the old doPollAsync + makeNotifyCallback machinery. - Producers now notify the node on a readiness transition: `PIPEFS` on writes, `SOCKFS` via its emit bridge, and `close()` wakes any waiter with `POLLNVAL`. - Socket hangup semantics tightened: a peer half-close is `POLLRDHUP` (only a fully closed connection is `POLLHUP`), and a queued client makes a listening socket `POLLIN`. The stream_ops.poll backend handler signature changes from `poll(stream, timeout)` to `poll(stream)` returning the current readiness mask. Note: the `exclusive` parameter on `addListener` and the round-robin wakeup path in `notifyListeners` are deliberately layered in here as groundwork for the EPOLLEXCLUSIVE handling that lands in the epoll followup (#27207). No caller passes `exclusive` in this PR yet, so that branch is currently inert — it's included now to keep the wait-queue API stable across the split.
|
This has been rebased to main, the diff should be much easier to review. |
sbc100
left a comment
There was a problem hiding this comment.
I wonder if all these tests added to test_core.py here should really go in test_other.py.
The issue with adding a lot of tests to test_core.py is that each test there is run in N different configurations, so they cost of a lot more. If we are just (or mostly) testing JS API functionality I think maybe test_other.py makes more sense.
| if not common.check_node_version(26): | ||
| self.node_args += ['--experimental-wasm-stack-switching'] | ||
| self.cflags += ['-Wno-experimental'] | ||
| self.set_setting('JSPI') |
There was a problem hiding this comment.
Why not just self.require_jspi() here (and above)?
There was a problem hiding this comment.
This broke CI, because the sockets tests are written as a browser environment (hangover from socket fs I believe), and require_jspI() early-exists when is_browser_test() is true. So we need to explicitly add the Node version checks instead and enable JSPI on Node.js 22 for them to work. Probably a better refactoring here, but for now I've retained this check, but abstracted as a new function at least.
5cef94c to
88282e3
Compare
Add epoll_create1/epoll_ctl/epoll_wait/epoll_pwait on the legacy (non-WASMFS) JS syscall layer, built on the per-inode readiness wait-queue: level- and edge-triggered modes, EPOLLONESHOT, EPOLLEXCLUSIVE, EPOLLRDHUP, nesting, and blocking waits under PROXY_TO_PTHREAD, ASYNCIFY, and JSPI. Also add emscripten_epoll_set_callback (new experimental <emscripten/epoll.h>), a non-blocking variant that delivers an epoll set's readiness to a JS callback without ASYNCIFY/JSPI.
Implements epoll, based on previously landed work #27226 and #27206.
Resolves #5033, #10556.
Adds
epoll_create1,epoll_ctl,epoll_wait,epoll_pwaitand a non-blocking JS-callback variant,emscripten_epoll_set_callback, on a single fd readiness model shared withpoll().Builds off of the existing event-driven readiness model in the JS FS system with the integration point as the per-inode wait-queue, having each FS node carrying a
listenersset and producers callingnotifyNodeListeners(node, flags)on ready transitions. There is no separate or parallel readiness machinery - it integrates directly with the existing model.pollOne(fd, events)is reused on the same readiness definition.Per standard epoll semantics -
epoll_ctl ADDinstalls a new listener on the watched node. If items are already ready they are added to the ready list. That listener then appends the registration to the epoll's ready list for waking. The epoll_wait consumes the ready list, re-checking each item against its current mask viapollOne.EPOLLONESHOTclears listeners to avoid unnecessary callback firing.EPOLL_CTL_MODcan then re-arm them again.EPOLLETis implemented correctly to avoid refiring items that remain readyEPOLLEXCLUSIVEis passed for listeners allowing only one wake for multiple epoll listeners to avoid the "thundering herd".maxevents, draining follows Linux-like semantics in supporting round-robin ready calling. To achieve this without losing performance, a doubly-linked list is used for the registrations. A simpler set / array with copying could be used alternatively if we don't want to use this approach.To support JS callbacks without JSPI/threads, a new
emscripten_epoll_set_callbackis implemented. This was implemented here to verify its comprehensive integration with all of the implemented epoll semantics, but could also be split out into a separate PR if necessary. It allows registering a persistent consumer on that same ready list as the epoll - the runtime delivers the ready set to the callback on each progress as if it were responding to anepoll_wait, but on the next tick after exiting the stack with no blocking and no ASYNCIFY/JSPI. It is armed once for the entire epoll, then consistently re-fires on the next tick while the set stays ready (so level and overflow drain as a blocking epoll_wait loop would). There is at most one callback per epoll (a second call replaces it; a NULL callback unregisters). Full integration with ready-list semantics work out naturally as it is just another consumer of the ready list.EPOLLET/EPOLLONESHOTEPOLLEXCLUSIVE/maxeventsall work out and apply to this callback design, so a single callback can fully integrate with normal epoll semantics.Most of the diff is tests, covering these semantics in depth including error handling, level versus edge reporting, nesting and ELOOP, fd-close auto-removal, JSPI and pthreads, real sockets, deregistration. For
emscripten_epoll_set_callbackcomprehensive tests are added for integrating with JSPI blockingepoll_waitin parallel and verifying both deterministically drain the same ready list with a wait and a callback on one epoll take disjoint slices rather than each seeing private or overlapping copies.Minor semantic divergences to note:
epoll_pwaitignoressigmaskepoll_create1ignoresEPOLL_CLOEXECepoll_eventunder Wasm in Musl is laid out as aligned 16 rather than x86-64's packed 12 bytes.PR made with AI assistance, under my review